Geometric, Semantic, and System-Level Scene Understanding for Improved Construction and Operation of the Built Environment

Recent advances in robotics and enabling fields such as computer vision, deep learning, and low-latency data passing offer significant potential for developing efficient and low-cost solutions for improved construction and operation of the built environment. Examples of such potential solutions include the introduction of automation in environment monitoring, infrastructure inspections, asset management, and building performance analyses. In an effort to advance the fundamental computational building blocks for such applications, this dissertation explored three categories of scene understanding capabilities: 1) Localization and mapping for geometric scene understanding that enables a mobile agent (e.g., robot) to locate itself in an environment, map the geometry of the environment, and navigate through it; 2) Object recognition for semantic scene understanding that allows for automatic asset information extraction for asset tracking and resource management; 3) Distributed coupling analysis for system-level scene understanding that allows for discovery of interdependencies between different built-environment processes for system-level performance analyses and response-planning. First, this dissertation advanced Simultaneous Localization and Mapping (SLAM) techniques for convenient and low-cost locating capabilities compared with previous work. To provide a versatile Real-Time Location System (RTLS), an occupancy grid mapping enhanced visual SLAM (vSLAM) was developed to support path planning and continuous navigation that cannot be implemented directly on vSLAM’s original feature map. The system’s localization accuracy was experimentally evaluated with a set of visual landmarks. The achieved marker position measurement accuracy ranges from 0.039m to 0.186m, proving the method’s feasibility and applicability in providing real-time localization for a wide range of applications. In addition, a Self-Adaptive Feature Transform (SAFT) was proposed to improve such an RTLS’s robustness in challenging environments. As an example implementation, the SAFT descriptor was implemented with a learning-based descriptor and integrated into a vSLAM for experimentation. The evaluation results on two public datasets proved the feasibility and effectiveness of SAFT in improving the matching performance of learning-based descriptors for locating applications. Second, this dissertation explored vision-based 1D barcode marker extraction for automated object recognition and asset tracking that is more convenient and efficient than the traditional methods of using barcode or asset scanners. As an example application in inventory management, a 1D barcode extraction framework was designed to extract 1D barcodes from video scan of a built environment. The performance of the framework was evaluated with video scan data collected from an active logistics warehouse near Detroit Metropolitan Airport (DTW), demonstrating its applicability in automating inventory tracking and management applications. Finally, this dissertation explored distributed coupling analysis for understanding interdependencies between processes affecting the built environment and its occupants, allowing for accurate performance and response analyses compared with previous research. In this research, a Lightweight Communications and Marshalling (LCM)-based distributed coupling analysis framework and a message wrapper were designed. This proposed framework and message wrapper were tested with analysis models from wind engineering and structural engineering, where they demonstrated the abilities to link analysis models from different domains and reveal key interdependencies between the involved built-environment processes.PHDCivil EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/155042/1/lichaox_1.pd

Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

R3^3-Net: A Deep Network for Multi-oriented Vehicle Detection in Aerial Images and Videos

Vehicle detection is a significant and challenging task in aerial remote sensing applications. Most existing methods detect vehicles with regular rectangle boxes and fail to offer the orientation of vehicles. However, the orientation information is crucial for several practical applications, such as the trajectory and motion estimation of vehicles. In this paper, we propose a novel deep network, called rotatable region-based residual network (R$^3$-Net), to detect multi-oriented vehicles in aerial images and videos. More specially, R$^3$-Net is utilized to generate rotatable rectangular target boxes in a half coordinate system. First, we use a rotatable region proposal network (R-RPN) to generate rotatable region of interests (R-RoIs) from feature maps produced by a deep convolutional neural network. Here, a proposed batch averaging rotatable anchor (BAR anchor) strategy is applied to initialize the shape of vehicle candidates. Next, we propose a rotatable detection network (R-DN) for the final classification and regression of the R-RoIs. In R-DN, a novel rotatable position sensitive pooling (R-PS pooling) is designed to keep the position and orientation information simultaneously while downsampling the feature maps of R-RoIs. In our model, R-RPN and R-DN can be trained jointly. We test our network on two open vehicle detection image datasets, namely DLR 3K Munich Dataset and VEDAI Dataset, demonstrating the high precision and robustness of our method. In addition, further experiments on aerial videos show the good generalization capability of the proposed method and its potential for vehicle tracking in aerial videos. The demo video is available at https://youtu.be/xCYD-tYudN0

DiffusionShield: A Watermark for Copyright Protection against Generative Diffusion Models

Recently, Generative Diffusion Models (GDMs) have showcased their remarkable capabilities in learning and generating images. A large community of GDMs has naturally emerged, further promoting the diversified applications of GDMs in various fields. However, this unrestricted proliferation has raised serious concerns about copyright protection. For example, artists including painters and photographers are becoming increasingly concerned that GDMs could effortlessly replicate their unique creative works without authorization. In response to these challenges, we introduce a novel watermarking scheme, DiffusionShield, tailored for GDMs. DiffusionShield protects images from copyright infringement by GDMs through encoding the ownership information into an imperceptible watermark and injecting it into the images. Its watermark can be easily learned by GDMs and will be reproduced in their generated images. By detecting the watermark from generated images, copyright infringement can be exposed with evidence. Benefiting from the uniformity of the watermarks and the joint optimization method, DiffusionShield ensures low distortion of the original image, high watermark detection performance, and the ability to embed lengthy messages. We conduct rigorous and comprehensive experiments to show the effectiveness of DiffusionShield in defending against infringement by GDMs and its superiority over traditional watermarking methods

Detecting GPC3-Expressing Hepatocellular Carcinoma with L5 Peptide-Guided Pretargeting Approach: An In Vitro MRI Experiment

Background and Aim: Glypican-3 (GPC3) is a novel molecular target for hepatocellular carcinoma (HCC). This study investigated the potential of an L5 peptide-guided pretargeting approach to identify GPC3-expressing HCC cells using ultra-small super-paramagnetic iron oxide (USPIO) as the MRI probe.Methods: Immunofluorescence with carboxyfluorescein (FAM)-labeled L5 peptide was performed in HepG2 and HL-7702 cells. Polyethylene glycol-modified ultrasmall superparamagnetic iron oxide (PEG-USPIO) and its conjugates with streptavidin (SA-PEG-USPIO) were synthesized, and hydrodynamic diameters, zeta potential, T2 relaxivity, and cytotoxicity were measured. MR T2-weighted imaging of HepG2 was performed to observe signal changes in the pretargeting group, which was first incubated with biotinylated L5 peptide and then with SA-PEG-USPIO. Prussian blue staining of cells was used to assess iron deposition.Results: Immunofluorescence assays showed high specificity of L5 peptide for GPC3. SA-PEG-USPIO nanoparticles had ≈36 nm hydrodynamic diameter, low toxicity, negative charge and high T2 relaxivity. MR imaging revealed that a significant negative enhancement was only observed in HepG2 cells from the pretargeting group, which also showed significant iron deposition with Prussian blue staining.Conclusion: MR imaging with USPIO as the probe has potential to identify GPC3-expressing HCC through L5 peptide-guided pretargeting approach

Backdoor Attacks on Crowd Counting

Crowd counting is a regression task that estimates the number of people in a scene image, which plays a vital role in a range of safety-critical applications, such as video surveillance, traffic monitoring and flow control. In this paper, we investigate the vulnerability of deep learning based crowd counting models to backdoor attacks, a major security threat to deep learning. A backdoor attack implants a backdoor trigger into a target model via data poisoning so as to control the model's predictions at test time. Different from image classification models on which most of existing backdoor attacks have been developed and tested, crowd counting models are regression models that output multi-dimensional density maps, thus requiring different techniques to manipulate. In this paper, we propose two novel Density Manipulation Backdoor Attacks (DMBA$^{-}$ and DMBA$^{+}$) to attack the model to produce arbitrarily large or small density estimations. Experimental results demonstrate the effectiveness of our DMBA attacks on five classic crowd counting models and four types of datasets. We also provide an in-depth analysis of the unique challenges of backdooring crowd counting models and reveal two key elements of effective attacks: 1) full and dense triggers and 2) manipulation of the ground truth counts or density maps. Our work could help evaluate the vulnerability of crowd counting models to potential backdoor attacks.Comment: To appear in ACMMM 2022. 10pages, 6 figures and 2 table

Boosting CO2 electrolysis performance : via calcium-oxide-looping combined with in situ exsolved Ni-Fe nanoparticles in a symmetrical solid oxide electrolysis cell

Financial support from National Key Research & Development Project (2016YFE0126900), National Natural Science Foundation of China (51672095), Hubei Province (2018AAA057) and the EPSRC Capital for Great Technologies Grant EP/L017008/1. We are grateful to the China Scholarship Council for funding (201806160178).The electrocatalysis of CO2 to valuable chemical products is an important strategy to combat global warming. Symmetrical solid oxide electrolysis cells have been extensively recognized for their CO2 electrolysis abilities due to their high efficiency, low cost, and reliability. Here, we produced a novel electrode containing calcium oxide-looping and in situ exsolved Ni–Fe nanoparticles by performing a one-step reduction of La0.6Ca0.4Fe0.8Ni0.2O3−δ (LCaFN). The CO2 captured by CaO was electrolyzed in situ by the Ni–Fe nanocatalysts. The cell with this special cathode showed a higher current density (0.632 A cm−2vs. 0.32 A cm−2) and lower polarization resistance (0.399 Ω cm2vs. 0.662 Ω cm2) than the unreduced LCaFN cathode at 800 °C with an applied voltage of 1.3 V. Use of the developed novel electrode offers a promising strategy for CO2 electrolysis.PostprintPeer reviewe